Multiple phase method for image deconvolution

ABSTRACT

A method for deblurring a blurry image ( 400 ) includes the steps of: performing a first phase of deconvolution ( 202 ) with a first phase regularization spatial mask ( 300 ) to reconstruct the main edges and generate a first phase latent sharp image ( 404 ) having reconstructed main edges; and performing a second phase of deconvolution ( 204 ) with a second phase regularization spatial mask ( 304 ) to reconstruct the texture and generate a second phase latent sharp image ( 406 ). The second phase regularization spatial mask ( 304 ) can be different from the first phase regularization spatial mask ( 300 ).

RELATED APPLICATION

This application claims priority on U.S. Provisional Application Ser.No. 61/859,740, filed Jul. 29, 2013 and entitled “MULTIPLE PHASE METHODFOR IMAGE DECONVOLUTION.” As far as permitted, the contents of U.S.Provisional Application Ser. No. 61/859,740 are incorporated herein byreference.

BACKGROUND

Cameras are commonly used to capture an image of a scene that includesone or more objects. Unfortunately, some of the images can be blurred.For example, movement of the camera and/or movement of the objects inthe scene during the exposure time of the camera can cause the image tobe blurred. Further, if the camera is not properly focused when theimage is captured, the image can be blurred.

Deconvolution methods used for deblurring tend to be very complex andslow (with the exception of the most simple methods that produce verylow quality images) and deblurred images tend to contain a lot of strongartifacts, most often in the form of ringing around edges in the image.Also, there is a tendency to strongly magnify noise. Techniques thathelp to reduce ringing artifacts and noise magnification in deconvolvedimages however tend to reduce any fine texture. Thus, the resultingimages have an unnatural, posterized look, as many areas in these imagesappear unnaturally smooth and the textures are blurry and muddled.

When blur is sufficiently spatially uniform, a blurred captured imagecan be modeled as the convolution of a latent sharp image with somepoint spread function (“PSF”) plus noise,B=K*L+N.  Equation(1).In Equation 1 and elsewhere in this document, (i) “B” represents ablurry image, (ii) “L” represents a latent sharp image, (iii) “K”represents a PSF kernel, and (iv) “N” represents noise (includingquantization errors, compression artifacts, etc.).

A non-blind deconvolution problem seeks to recover the latent sharpimage L only when the PSF kernel K is known. Alternatively, a blinddeconvolution problem seeks to recover both the PSF kernel K and thelatent sharp image L. Both the blind deconvolution problem and non-blinddeconvolution problem are difficult to accurately solve because they areill conditioned, and some information has been irretrievably lost due tozero and near zero values in the corresponding optical transfer function(“OTF”), Fourier transform of the PSF.

Moreover, many blurry images include areas that further complicate theproblem of recovering a latent sharp image L. For example, extremelybright areas where the sensor pixels reach their saturation point in theblurry image B can adversely influence the determined PSF kernel K andthe resulting latent sharp image L.

One common approach to solving a deconvolution problem includesreformulating it as an optimization problem in which suitable costfunctions are minimized in order to find a solution to the deconvolutionproblem.

A relatively common type of a cost function used for deconvolution is aregularized least squares cost function. Typically, a regularized leastsquares cost function consists of (i) one or more fidelity terms, whichmake the minimum conform to equation (1) modeling of the blurringprocess, and (ii) one or more regularization terms, which make thesolution more stable and help to enforce prior information about thesolution, such as sparseness.

An example of a generalization of a latent sharp image cost function isc(L)=∥K*L−B∥ _(p) ^(p)+ω∥(D _(x) *L)∥_(q) ^(q)+∥(D _(y) *L)∥_(q)q).  Equation(2)In Equation (2), and in all other formulas in this document (i) thesubscript p denotes the norm for the fidelity term, (ii) the superscriptp denotes the power for the fidelity term, (iii) the subscript q denotesthe norm for the regularization terms, (iv) the superscript q denotesthe power for the regularization terms, and (v) D_(x) and D_(y) denotethe convolution kernels used for computing derivatives in x- andy-direction, respectively. Equation 2 is a regularized least squarescost function when p is equal to two. The first term in Equation (2),∥K*L−B∥_(p) ^(p), is the fidelity term that forces the solution tosatisfy blurring model in Equation (1) with a noise term that is small.Further, in Equation (2), ∥(D_(x)*L)∥_(q) ^(q) and ∥(D_(y)*L)∥_(q) ^(q)are regularization terms that help to infuse prior information aboutarrays that can be considered a latent sharp image. The derivatives of anatural image tend to be sparse: in most points derivatives are close tozero and only in relatively few points (edge points) the values arelarge. The regularization terms help to enforce this property. InEquation (2), ω is a regularization weight, a selected constant thathelps to achieve the proper balance between the fidelity term and theregularization terms. It should be noted that this example represents avery simple cost function. Often, more complicated cost functions areused, such as those with more fidelity and regularization termsinvolving derivatives of 1^(st) or higher order.

When the image derivatives are assumed to follow a Guassian priordistribution, the power (superscript) and norm (subscript) for theregularization term(s) in the cost function is equal to two (q=2) (alsoreferred to as a “Gaussian prior” or “2-norm”), and the cost function isoften referred to as a Gaussian prior cost function. Unfortunately, theresulting latent sharp image from a Gaussian prior cost function ofteninclude strong ringing artifacts in the vicinity of high contrast edges.

In an attempt to improve the resulting latent sharp image, other powersand norms have been tried in the regularization term(s). For example,when image derivatives are assumed to follow a Laplacian priordistribution, the resulting regularization term in the cost function hasa power and a norm that are equal to one (q=1) (also referred to as a“Laplacian prior” or a “1-norm”); and when image derivatives are assumedto follow a hyper-Laplacian prior distribution, the resultingregularization term in the cost function has a power and a norm that areequal to less than one (q<1) (also referred to as a “hyper-Laplacianprior” or “q-pseudonorm”). With a Laplacian or hyper-Laplacian priorcost function, the ringing artifacts are reduced, but the fine texturein the image is often not recovered and the image does not look natural.

SUMMARY

The present invention is directed to a method for deblurring a blurryimage with a known or estimated point spread function, the blurry imageincluding main edges and texture. In one embodiment, the method includesthe steps of: performing a first phase of deconvolution with a firstphase regularization spatial mask to reconstruct the main edges andgenerate a first phase latent sharp image having reconstructed mainedges; and performing a second phase of deconvolution with a secondphase regularization spatial mask to reconstruct the texture andgenerate a second phase latent sharp image, the second phaseregularization spatial mask being different from the first phaseregularization spatial mask. As provided herein, the proposed multiplephase approach provided herein produces high quality deblurred images.

In one embodiment, the second phase regularization spatial mask isgenerated using the first phase latent sharp image. Further, as providedherein, (i) the first phase of regularization can include a first phasealgorithm that assumes a Laplacian or Hyper-Laplacian prior and includesthe first phase regularization spatial mask to suppress ringingartifacts and noise, and (ii) the second phase of regularization caninclude a second phase algorithm that assumes a Gaussian prior, andincludes a spatial prior, and the second phase regularization spatialmask to help to recover image texture without introducing strong ringingartifacts.

As used herein, an assumption of a Laplacian prior shall mean a power ofone (q=1) in regularization term(s), an assumption of a hyper-Laplacianprior shall mean a power of less than one (q<1) in regularizationterm(s), and an assumption of a Gaussian prior shall mean a power thatis equal to two (q=2).

Additionally, the method can include the steps of (i) identifying anyoutlier regions in the blurry image, and (ii) creating a fidelity termmask based on the identified outlier regions. Further, the first phasealgorithm can use the fidelity term mask, and the second phase algorithmcan use the fidelity term mask.

Moreover, the method can include the steps of (i) performing a thirdphase of deconvolution on the blurry image with a third phaseregularization spatial mask to reconstruct the main edges to generate athird phase latent sharp image; and (ii) performing a fourth phase ofdeconvolution with a fourth phase regularization spatial mask toreconstruct the texture to generate a fourth phase latent sharp image,the fourth phase regularization spatial mask being different from thethird phase regularization spatial mask. In this embodiment, the methodcan include the step of merging the second phase latent sharp image withthe fourth phase latent sharp image to create the output latent sharpimage.

Further, the method can include the steps of (i) converting the blurryimage to a color space that includes a luminance channel and chrominancechannels, (ii) performing deconvolution of the luminance channel of theblurry image with the first phase and the second phase of deconvolution,(iii) generating the second phase regularization spatial mask using thedeblurred luminance channel, and (iv) performing deconvolution of thechrominance channels of the blurry image with the first phase and thesecond phase of deconvolution.

In another embodiment, the method includes the steps of: (i) performinga first phase of deconvolution with a first latent sharp image costfunction that assumes a Laplacian or Hyper-Laplacian priorregularization to generate a first phase latent sharp image havingreconstructed main edges; and (ii) performing a second phase ofdeconvolution with a second latent sharp image cost function thatassumes a Gaussian prior regularization and a spatial priorregularization to generate a second phase latent sharp image withreconstructed texture.

In still another embodiment, the present invention is directed to asystem for deblurring a blurry image that includes a control system that(i) performs a first phase of deconvolution with a first phaseregularization spatial mask to reconstruct the main edges and generate afirst phase latent sharp image having reconstructed main edges; and (ii)performs a second phase of deconvolution with a second phaseregularization spatial mask to reconstruct the texture and generate asecond phase latent sharp image, the second phase regularization spatialmask being different from the first phase regularization spatial mask.

In yet another embodiment, the present invention, the control system (i)performs a first phase of deconvolution with a first latent sharp imagecost function that assumes a Laplacian or Hyper-Laplacian priorregularization to generate a first phase latent sharp image havingreconstructed main edges; and (ii) performs a second phase ofdeconvolution with a second latent sharp image cost function thatassumes a Gaussian prior regularization and a spatial priorregularization to generate a second phase latent sharp image withreconstructed texture.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of this invention, as well as the invention itself,both as to its structure and its operation, will be best understood fromthe accompanying drawings, taken in conjunction with the accompanyingdescription, in which similar reference characters refer to similarparts, and in which:

FIG. 1 is a simplified view of a camera and a computer having featuresof the present invention;

FIG. 2 illustrates a first embodiment of a multi-phase imagedeconvolution method having features of the present invention;

FIG. 3A is a simplified illustration of a portion of a Subphase 1Aregularization spatial mask N₁ having features of the present invention;

FIG. 3B is a simplified illustration of a portion of a Subphase 1Bregularization spatial mask N₁ having features of the present invention;

FIG. 3C is a simplified illustration of a portion of a Phase 2regularization spatial mask N₂ having features of the present invention;

FIG. 4A is an illustration of a blurry image;

FIG. 4B is an illustration of a latent sharp image after Subphase 1Aprocessing of the blurry image of FIG. 4A;

FIG. 4C is an illustration of a latent sharp image after Subphase 1Bprocessing of the blurry image of FIG. 4A;

FIG. 4D is an illustration of a final latent sharp image after Phase 2processing of the blurry image of FIG. 4A;

FIG. 4E is an illustration of a true PSF of the blurry image of FIG. 4A;

FIG. 4F is an illustration of an estimated PSF of the blurry image ofFIG. 4A;

FIG. 5A illustrates another multi-phase image deconvolution methodhaving features of the present invention;

FIG. 5B is a simplified illustration of a portion of a fidelity spatialmask M having features of the present invention;

FIG. 6A illustrates a blurry image;

FIG. 6B illustrates a Phase 2 latent sharp image;

FIG. 6C illustrates a final latent sharp image generated by the presentinvention;

FIG. 7 is an algorithm flowchart for a multi-phase, non-blinddeconvolution method;

FIG. 8A is a simplified illustration of a Phase 2 luminance latent sharpimage (L_(Ph2) _(_) _(Lum)), a Phase 4 luminance latent sharp image(L_(Ph4) _(_) _(Lum)), and a luminance channel latent sharp image(L_(Lum));

FIG. 8B is a simplified illustration of a Phase 2 Cb chrominance latentsharp image (L_(Ph2) _(_) _(CbChr)), a Phase 4 Cb chrominance latentsharp image (L_(Ph4) _(_) _(CbChr)), and a Cb chrominance channel latentsharp image (L_(CbChr));

FIG. 8C is a simplified illustration of a Phase 2 Cr chrominance latentsharp image (L_(Ph2) _(_) _(CrChr)), a Phase 4 Cr chrominance latentsharp image (L_(Ph4) _(_) _(CrChr)), and a Cr chrominance channel latentsharp image (L_(CrChr)); and

FIG. 8D is a simplified illustration of the luminance channel latentsharp image (L_(Lum)), the Cb chrominance channel latent sharp image(L_(CbChr)), the Cr chrominance channel latent sharp image (L_(CrChr))and a final latent sharp image (L_(Final)).

DESCRIPTION

FIG. 1 is a simplified perspective view of an image apparatus 10 (e.g. adigital camera), and a computer 12 (illustrated as a box). It should benoted that the image apparatus 10 and/or the computer 12 can be referredto as a system. As is known, movement of the image apparatus 10, and/ormovement of an object (not shown in FIG. 1) in the scene (not shown)during the capturing of an image (not shown in FIG. 1) can cause imageblur. Additionally, or in the alternative, blur in the image can becaused by other things, such as, the image apparatus 10 not beingproperly focused when the image is captured.

As an overview, in certain embodiments, the present invention isdirected to a number of multiple phase algorithms for estimating thelatent sharp image of a blurred image and deconvoluting a blurred image.Stated in another fashion, the present invention is directed to one ormore, multiple phase, non-blind image deconvolution methods thatefficiently suppress ringing artifacts and noise, and recover imagetexture. In certain embodiments, the proposed methods significantlyimprove the quality of deblurred images by using one or more of thefollowing features (i) multiple phases of deconvolution with differentadaptive regularization (L1 Laplace or Hyper-Laplace) priorregularization and combined L2 Gaussian and spatial priorregularization; (ii) separate deconvolution of highlight andnon-highlight regions by using spatial mask in fidelity term; and/or(iii) separate deconvolution of luminance and chrominance channels withdeblurred luminance channel being used to produce adaptiveregularization masks for chrominance channels. As a result thereof, theproposed multiple phase approaches provided herein produce high qualitydeblurred images.

In one embodiment, as provided herein, the image apparatus 10 caninclude a capturing system 13 that captures the image, and a controlsystem 14 that uses one or more of the algorithms for deconvoluting theblurred image for in camera processing. Alternatively, the computer 12can include a control system 16 that uses one or more of the algorithmsfor deconvoluting the blurred image. In either event, the control system14, 16 provides a deblurred latent sharp image from the blurred image.Each control system 14, 16 can include one or more processors andcircuits. Further, either of the control systems 14, 16 can includesoftware that utilizes one or more methods (performs the algorithms)provided herein to deblur the blurry image.

The methods provided herein help to significantly improve the quality ofdeblurred images. The resulting deblurred images contain relatively fewringing artifacts and noise, and more fine texture. Further, in certainembodiments, more difficult images can be handled successfully becausecolor ringing artifacts and heavy ringing artifacts around clippedhighlight regions are significantly reduced.

FIG. 2 illustrates a first embodiment of a multi-phase imagedeconvolution method. In one embodiment, the multi-phase imagedeconvolution method can be used for non-blind deconvolution with aknown or estimated point spread function. It should be noted that theorder of some of the steps in FIG. 2 can be switched, performedconcurrently, and/or some of the steps can be optional. In thisembodiment, the multi-phase image deconvolution method is presented as atwo phase method. At block 200, the blurred image is read by thealgorithm. Subsequently, Phase 1 processing of the blurred image isperformed at block 202 to restore and reconstruct the main edges with astrong emphasis on suppressing ringing. Next, Phase 2 processing isperformed at block 204 to restore texture. Upon completion of Phase 2processing, the output Latent Sharp Image is provided at block 205. Asprovided herein, because of the two phase method, the resultingdeblurred image contains relatively few ringing artifacts and noise, andmore fine texture.

Referring to FIG. 2, at block 204, an adaptive phase 1 regularizationmask N₁ is generated, and at block 206, a phase 1 regularization weightω₁ is selected. The adaptive regularization mask N₁ can be coarse andcan be computed from the blurry image. The regularization weight ω₁ isselected to achieve the desired amount of regularization. Non-exclusiveexamples of a suitable regularization weight ω₁ are approximately 1 e-1,1 e-2, 1 e-3, or 1 e-4, the value of which will depend on the accuracyof the PSFs and other factors. Subsequently, at block 208, a phase 1latent sharp image cost function is minimized. As provided herein, incertain embodiments, the phase 1 latent sharp image cost function uses aLaplacian prior or a Hyper-Laplacian prior, and the phase 1 adaptiveregularization mask N₁.

As provided herein, the phase 1 latent sharp image cost function can beminimized multiple times, with each subsequent minimization beingperformed with an improved regularization mask N₁ and decreasedregularization (smaller regularization weight ω₁). For ease ofdiscussion, the first minimization of the phase 1 latent sharp imagecost function can be referred to as Subphase 1A processing, and thesecond minimization of the phase 1 latent sharp image cost function canbe referred to as Subphase 1B processing.

In certain instances, Subphase 1A and Subphase 1B processing is all thatis necessary to accurately reconstruct the main edges. Alternatively, ifadditional processing is desired to better restore the edges, a thirdminimization of the phase 1 latent sharp image cost function can bereferred to as Subphase 1C processing, and subsequent minimizations canbe referred to as Subphase 1D, 1E, etc.

Typically, during Subphase 1A, the phase 1 cost function can beminimized by variable splitting and alternating minimization inapproximately 6-10 iterations. During Subphase 1A processing, there is acoarse restoration of image with the strong emphasis on suppressingringing, and the Subphase 1A latent sharp image is slightly blurry imagewith very little ringing.

After Subphase 1A processing is complete, at block 210 the number ofminimizations is evaluated to determine if the desired number ofminimizations has been performed. If the desired number of minimizationsis two, then the answer is no and Subphase 1B processing begins. InSubphase 1B processing, at block 212, the phase 1 regularization mask N₁is updated and improved, and at block 214, the regularization isdecreased by decreasing the phase 1 regularization weight ω₁. Forexample, in Subphase 1B, the phase 1 regularization mask N₁ can becomputed from both the blurry image and the Phase 1A latent sharp image.Further, as a non-exclusive example, the phase 1 regularization weightω₁ can be approximately one, two, five, seven, or ten times less foreach subsequent Subphase.

Next, at block 208, the phase 1 latent sharp image cost function isagain minimized with the improved mask N₁ and decreased regularization.Typically, during Subphase 1B, the phase 1 cost function can beminimized by variable splitting and alternating minimization inapproximately 6-10 iterations. After Subphase 1B processing, there ismore accurate restoration of the image with strong emphasis onsuppressing ringing. Typically, the Subphase 1B algorithm produces aSubphase 1B Latent sharp image with sharp main edges and very littleringing, but (in certain images) without fine texture due to strongringing suppression by the Laplacian or Hyper-Laplacian prior.

After Subphase 1B processing, at block 210 the number of minimizationsis evaluated to determine if the desired number of minimizations hasbeen performed? If the desired number of minimizations is two, then theanswer is yes, Phase 1 processing is complete, the Phase 1B latent sharpimage is the Phase 1 latent sharp image, and Phase 2 processing begins.Alternatively, if the desired number is three or more, Subphase 1Cprocessing is performed with an updated phase 1 regularization mask N₁,and a further decreased phase 1 regularization weight ω₁. This loop canbe repeated until the desired the level of edge restoration hasoccurred.

Phase 2 processing begins at block 216 where an adaptive phase 2regularization mask N₂ is generated, and at block 218 where a phase 2regularization weight ω₂ is selected to achieve the desired level ofregularization. For example, in Phase 2, the phase 2 regularization maskN₂ can be computed from both the blurry image and the final result ofPhase 1 processing (the Phase 1 latent sharp image). Subsequently, atblock 220, a phase 2 latent sharp image cost function is minimized. Asprovided herein, in certain embodiments, the phase 2 latent sharp imagecost function can use the phase 2 regularization mask N₂, a Gaussianprior to recover and restore image texture, and a spatial prior toinhibit the introduction of strong ringing artifacts. The Phase 2 costfunction can be minimized by variable splitting and alternatingminimization (typically within approximately 3 iterations, for example).

After finishing the minimization of the Phase 2 cost function, theresulting latent sharp image is output at block 205. Alternatively, theminimization of the Phase 2 cost function can be repeated one or moretimes (somewhat similar to Phase 1) with each subsequent process usingan improved phase 2 regularization mask N₂.

With the present invention, (i) in Phase 1, the algorithm deconvolutesthe blurry image while efficiently suppressing ringing artifacts andnoise, and (ii) in Phase 2, the algorithm recovers the image texture.With this design, the multiple stage algorithm can produce deblurredimages that are almost free of ringing, contain very little noise, andalso contain natural texture. Further, in certain embodiments, themethod is very fast because variable splitting and alternatingminimization can be used to minimize the cost functions.

It should be noted that the two phase algorithm described above can alsobe described as a three phase algorithm where Subphase 1A and Subphase1B (which both use a Laplacian or Hyper-Laplacian prior) are consideredtwo separate phases.

As one non-exclusive example, the Phase 1 latent sharp image costfunction (used for Subphase 1A, 1B, etc., processing) can have thefollowing form for non-blind deconvolution:c(L)=α₁ ∥K*L−B∥ _(p) ^(p)+α₂(∥K*D _(x) *L−D _(x) *B∥ _(p) ^(p) +∥K*D_(y) *L−D _(y) *B∥ _(p) ^(p))+ω₁(∥N ₁(D _(x) *L)∥_(q) ^(q) +∥N ₁(D _(y)*L)∥_(q) ^(q).  Equation (3).

In Equation 3, and elsewhere in this document, (i) c(L) is the latentsharp image estimation cost function, (ii) L is the latent sharp image,(iii) K is the PSF kernel, (iv) B is the blurry image, (v) D_(x) andD_(y) are the first order partial derivative operators, (vi) alpha one“α₁” is a first fidelity weight parameter, and alpha two “α₂” is asecond fidelity weight parameter that help to set the proper balancebetween the fidelity terms to achieve the best compromise between thefidelity terms; (vii) ω₁ is the first regularization weight for thesmooth areas; (viii) the subscript p denotes the norm for the fidelityterm(s) and the superscript p denotes the power for the fidelityterm(s); (ix) the subscript q denotes the norm for the regularizationterm(s) and the superscript q denotes the power for the regularizationterm(s); and (x) N₁ is the first phase regularization spatial mask forPhase 1 processing. As provided herein, in Phase 1 processing, q isequal to one (q=1) if a Laplacian prior is assumed, or q is less thanone (q<1) if a hyper-Laplacian prior is assumed.

For Equation 3, other fidelity term(s) are possible. For example, it ispossible to use just one fidelity term (α₂=0). However, in certainembodiments, including fidelity terms with 1^(st) order derivatives andputting more weight on them (second fidelity weight parameter greaterthan the first fidelity weight parameter (α₂>>α₁)) yields betterresults. In certain embodiments, the fidelity term without derivativesmust be present (α₁>0) because all the regularization terms includederivatives and there would be zeros in the denominator of the closedform formula used for updating L otherwise. In one, non-exclusiveembodiment, the first fidelity weight parameter α₁ can be set to beequal to 0.001 (α₁=0.001), and (ii) the second fidelity weight parameterα₂ can set to be equal to one (α₂=1). However, other values can be used.With this design, the cost function of Equation (3) relies mainly onfidelity terms with derivatives. Using one or more fidelity terms withimage derivatives help to reduce ringing.

Equation (3) can be re-written with the power and norm for the fidelityterm(s) equal to two (p=2), and for the assumption of a Laplacian prior(q=1) as follows:c(L)=α₁ ∥K*L−B∥ ₂ ²+α₂(∥K*D _(x) *I−D _(x) *B∥ ₂ ² +∥K*D _(y) *L−D _(y)*B∥ ₂ ²)+ω₁(∥N ₁(D _(x) *L)∥₁ ¹ +∥N ₁(D _(y) *L)|₁ ¹).  Equation (4).

In one, non-exclusive embodiment, Equation (4) can be minimized byvariable splitting and alternating minimization. In certain cases,approximately six to ten iterations may be needed to suppress ringing.In certain embodiments, to minimize the cost function of Equation (4),the following function with auxiliary variables W_(x) and W_(y) can becreated:c(L,W _(x) ,W _(y))=α₁ ∥K*L−B∥ ₂ ²+α₂(∥K*D _(x) *B∥ ₂ ² +∥K*D _(y) *L−D_(y) *B∥ ₂ ²)+ω₁(∥N ₁ W _(x)∥₁ ¹ +∥N ₁ W _(y)∥₁ ¹+β(∥W _(x) −D _(x) *L∥₂ ² +W _(y) −D _(y) *L∥ ₂ ²)).  Equation (5)

In Equation (5), (i) auxiliary array W_(x) has replaced partialderivative D_(x)*L and auxiliary array W_(y) has replaced partialderivative D_(y)*L in the regularization terms; (ii) there are twopenalty terms, ∥W_(x)−D_(x)*L∥₂ ² and ∥W_(y)−D_(y)*L)∥₂ ² which forceW_(x)≈D_(x)*L and W_(y)≈D_(y)*L; and (iii) β is a penalty weight thathas been added to help balance the influence of the penalty terms.Equation (5) can be minimized iteratively, by alternating minimizationover W_(x), W_(y), and L, respectively. In all three cases, a closedform formula for the minimum exists that can be easily evaluated. Incertain embodiments, it is common to increase the penalty weight βduring iterating minimization over W_(x), W_(y), and L in alternatingfashion. In one, non-exclusive embodiment, the penalty weight of β=1 canbe used in a first iteration, and the penalty weight β can increased bya factor (e.g. a factor of 2 in one non-exclusive example) in eachsubsequent iteration.

The following minimization formulas can be used for W_(x) and W_(y),

$\begin{matrix}{{W_{x} = {{sthr}\left( {{D_{x}^{*}L},\frac{N}{2\beta}} \right)}},} & {{Equation}\mspace{14mu}(6)} \\{{W_{y} = {{sthr}\left( {{D_{y}^{*}L},\frac{N}{2\beta}} \right)}},} & {{Equation}\mspace{14mu}(7)}\end{matrix}$sthr(x;y)=sign(x)max(0,|x|−y)  Equation (8).

In Equation (8), sthr is the soft thresholding function (applied to eachelement of the array).

As provided herein, the closed form formula for the minimum of thelatent sharp image cost function of Equation (5) is as follows:

$\begin{matrix}{{F(L)} = {\frac{\begin{matrix}{{\left( {{\alpha 1} + {{\alpha 2}\; D}} \right)\overset{\_}{F(K)}{F(B)}} +} \\{\omega_{1}{B\left( {{\overset{\_}{F\left( D_{x} \right)}{F\left( W_{x} \right)}} + {\overset{\_}{F\left( D_{y} \right)}{F\left( W_{y} \right)}}} \right)}}\end{matrix}}{{\left( {{\alpha 1} + {{\alpha 2}\; D}} \right)\overset{\_}{F(K)}{F(K)}} + {\omega_{1}\beta\; D}}.}} & {{Equation}\mspace{14mu}(9)}\end{matrix}$

In Equation (9), F is the Fourier transform operator, and D has thefollowing value:D= F(D _(x)) F(D _(x))+ F(D _(y)) F(D _(y))  Equation (10).

The regularization spatial mask N₁ for Phase 1 (and N₂ for Phase 2) isan array of individual weights for individual pixels in theregularization term(s) that can be used to suppress noise and ringing.As a non-exclusive example, the pixels in the regularization spatialmask can have any value between 0 and 1. Alternatively, theregularization spatial mask can be a binary mask in which the pixelshave a value of either 0 and 1. Multiplication by the mask ispoint-wise.

There is a number of possible ways of creating the regularizationspatial masks. In one non-exclusive example, the estimation of theregularization spatial mask N₁ in Phase 1 is performed at least twice,(i) first (for Subphase 1A processing) with a coarse mask computed fromthe blurry original image, and (ii) second (for Subphase 1B processing)with a finer mask that is computed using the latent sharp image Lobtained after Subphase 1A processing, as well as the blurry image B.

For example, for Subphase 1A processing, the regularization spatial maskN₁ can have higher weights for the pixels in the smooth regions in theoriginal blurry image, and lower weights for the pixels in edge regionsof the original blurry image. For example, a high pass filter can beused on the original blurry image to identity the pixels in the smoothregions and the pixels in the edge regions. FIG. 3A is a simplifiedillustration of a portion of a Subphase 1A regularization spatial maskN₁ 300 that has different values for the pixels.

Further, for Subphase 1B processing, the regularization spatial mask N₁can be computed using both the blurry image and the latent sharp image Lobtained from Subphase 1A processing. In one embodiment, theregularization spatial mask N₁ for Subphase 1B processing will havehigher weights for the pixels in the smooth regions, and lower weightsfor the pixels in edge regions. For example, a high pass filter can beused on both the original blurry image and the latent sharp image Lobtained from Subphase 1A processing to identity the pixels in thesmooth regions and the pixels in the edge regions. Thus, theregularization spatial mask N₁ used in the Subphase 1B processing willbe more accurate than the regularization spatial mask N₁ used in theSubphase 1A processing. FIG. 3B is a simplified illustration of aportion of a Subphase 1B regularization spatial mask N₁ 302 that hasdifferent values for the pixels.

In certain embodiments, the Subphase 1B cost function is the same as theSubphase 1A cost function (e.g. Equation 3), except weakerregularization is used (smaller value of regularization weight ω₁) andthe improved First Phase spatial mask N₁. As a non-exclusive example,with the weaker regularization of Subphase 1B, the value of theregularization weight ω₁ can be approximately five, ten, fifteen, ortwenty times weaker than the corresponding value in Phase 1A.

Additionally, as provided above, Phase 1 can include a Subphase 1Cprocessing (or more processing, e.g. Subphase 1D, Subphase 1E, etc) withthe same cost function (e.g. Equation 3), with each successive Subphaseprocessing including weaker regularization (smaller values ofregularization weight ω₁) and/or an improved regularization mask N₁based on the latent sharp image from the previous processing step.

Phase 1 produces images that are relatively clean, free of ringing andnoise, but they lack texture. In contrast, Phase 2 concentrates onrecovering the texture. To do that, in certain embodiments, the Phase 2algorithm (i) assumes a Gaussian prior (q=2) that does not suppresstexture as much as a Laplacian or hyper-Laplacian prior, (ii) a phase 2regularization spatial mask N₂ in the Gaussian prior regularizationterm, and (iii) a second prior based on the result of Phase 1 tosuppress ringing. One, non-exclusive example of a suitable Phase 2latent sharp image cost function has the following form:c(L)=α₁ ∥K*L−B∥ ² ₂ ²+α₂(∥K*D _(x) *L−D _(x) *B∥ ₂ ² +∥K*D _(y) *L−D_(y) *B∥ ₂ ²)+ω₂(∥N ₂(D _(x) *L)∥₂ ² +∥N ₂(D _(y) *L)∥₂ ²)+σ∥L−L_(Phase1)∥₂ ².  Equation (11)

In Equation (11), (i) ω₂ is the phase 2 regularization weight for thesmooth areas; (ii) N₂ is the phase 2 regularization spatial mask forPhase 2 processing; (iii) L_(Phase1) denotes the latent sharp imageestimated at the end of Phase 1 processing (e.g. output of Subphase 1B);(iv)∥L−L_(Phase1)∥₂ ² is a spatial prior; and (v) σ is a regularizationweight for the spatial prior to provide proper weight for the spatialprior. In Equation (11), (i) the square is used in the power and norm(q=2) of Regularization term(s) of the formula because the assumption ofa Gaussian prior is better for restoring texture; and (ii) the spatialprior is used to suppress ringing and noise.

For Phase 2 processing, a variable splitting technique can be used forsplitting and alternating minimization of the cost function of Equation(11) in an efficient manner. In this example, even though the Gaussianprior is assumed, the spatial masks N₂ in the regularization terms makeit impossible to derive a closed form formula for the minimum. The newcost function for Phase 2 derived from Equation (11) can have thefollowing form:c(L,W _(x) ,W _(y))=α₁ ∥K*L−B∥ ₂ ²+α₂(∥K*D _(x) *L−D _(x) *B∥ ₂ ² +∥K*D_(y) *L−D _(y) *B∥ ₂ ²)+ω₂(∥N ₂ W _(x)∥₂ ² +∥N ₂ W _(y)∥₂ ²+β(∥W _(x) −D_(x) *L∥ ₂ ² +∥W _(y) −D _(y) *L∥ ₂ ²))+σ∥L−L _(Phase1)∥₂ ²  Equation(12).

In Equation (12), in one, non-exclusive embodiment, the penalty weightof β=1 can be used in a first iteration, and the penalty weight β can beincreased by a factor (e.g. a factor of 2 in one non-exclusive example)in each subsequent iteration. As a non-exclusive example, a suitableregularization weight σ can be between approximately 1 e-2 and 1 e-3.However, other values for β and σ can be used.

The cost function of Equation (12) can be minimized by variablesplitting and alternating minimization over L, W_(x) and W_(y). In manycases, not many iterations seem to be needed. The formulas for updatingL, W_(x), and W_(y), and the cost function can be expressed aftersplitting as follows:

$\begin{matrix}{{W_{x} = \frac{D_{x}^{*}L}{1 + {\frac{1}{\beta}\left( {N\; 2} \right)^{2}}}},} & {{Equation}\mspace{14mu}(13)} \\{{W_{y} = \frac{D_{y}^{*}L}{1 + {\frac{1}{\beta}\left( {N\; 2} \right)^{2}}}},} & {{Equation}\mspace{14mu}(14)}\end{matrix}$

As provided herein, there is a closed form formula for the minimum ofthe latent sharp image cost function of Equation (12) as follows:

$\begin{matrix}{{F(L)} = {\frac{\begin{matrix}{{\left( {{\alpha 1} + {{\alpha 2}\; D}} \right)\overset{\_}{F(K)}{F(B)}} +} \\{{\sigma\;{F\left( L_{{Phase}\; 1} \right)}} + {\omega_{2}{\beta\left( {{\overset{\_}{F\left( D_{x} \right)}{F\left( W_{x} \right)}} + {\overset{\_}{F\left( D_{y} \right)}{F\left( W_{y} \right)}}} \right)}}}\end{matrix}}{{\left( {{\alpha 1} + {{\alpha 2}\; D}} \right)\overset{\_}{F(K)}{F(K)}} + {\omega_{2}\beta\; D}}.}} & {{Equation}\mspace{14mu}(15)}\end{matrix}$where D is the same as provided above in Equation (10).

In Phase 2, the regularization spatial mask N₂ can again be used tosuppress noise and ringing and can be computed from both the blurryimage and the latent sharp image estimated at the end of Phase 1processing (e.g. end of Subphase 1B). In one embodiment, theregularization spatial mask N₂ will have higher weights for the pixelsin the smooth regions, and lower weights for the pixels in edge regions.For example, a high pass filter can be used on both the original blurryimage and the result of Phase 1 processing to identity the pixels in thesmooth regions and the pixels in the edge regions. The spatial mask N₂used in the Phase 2 processing should be more accurate than the finalspatial mask N₁ used in the Phase 1 processing. FIG. 3C is a simplifiedillustration of a portion of a Phase 2 regularization spatial mask N₂304 that illustrates different values for different pixels.

FIG. 4A is an illustration of a blurry image 400. FIG. 4B is anillustration of a latent sharp image 402 after Subphase 1A processing ofthe blurry image 400 of FIG. 4A. After Subphase 1A processing, there isa coarse restoration of image with the strong emphasis on suppressingringing, and the Subphase 1A latent sharp image 402 is slightly blurryimage with very little ringing. FIG. 4C is an illustration of a latentsharp image 404 after Subphase 1B processing of the blurry image 400 ofFIG. 4A. The Subphase 1B processing produces a latent sharp image 404with very little ringing or noise, but the image lacks fine texture(looks posterized). FIG. 4D is an illustration of a final latent sharpimage 406 after Phase 2 processing of the blurry image 400 of FIG. 4A.The latent sharp image 406 produced after Phase 2 processing restorestexture to achieve a more natural looking restored image. Thus, asprovided herein, the multiple phase algorithm helps to obtain highquality deblurred images that are almost free of ringing artifacts andnoise, yet they do include fine texture.

FIG. 4E is an illustration of a true PSF 408 of the blurry image 400 ofFIG. 4A. The true PSF 408 is only known if an experiment is set up andthe blurry image 400 is artificially blurred with the true PSF 408.

FIG. 4F is an illustration of an estimated PSF 410 of the blurry image400 of FIG. 4A. The estimated PSF 410 can be estimated using a blinddeconvolution algorithm. Subsequently, the PSF 410 can be used in one ofthe non-blind deconvolution methods provided herein to obtain the entirefinal latent sharp image.

It should be noted that the Phase 1 and Phase 2 cost functions providedabove are non-exclusive examples, and that significant modifications arepossible for the Phase 1 and Phase 2 cost functions. For example, thePhase 1 and Phase 2 cost functions can be modified to (i) assumeLaplacian norm in the Fidelity term(s) (p=1); (ii) assume ahyper-Laplacian (pseudo-norm) in the Fidelity term(s) (p<1); and/or(iii) use a spatial mask in the fidelity term(s). Such cost functionscan be minimized using variable splitting algorithm similar to thatprovided above. Another non-exclusive example of a variable splittingtechnique that can be used is described in the paper by Ŝroubek andMilanfar (F. Ŝroubek, P. Milanfar: Robust Multichannel BlindDeconvolution via Fast Alternating Minimization, IEEE Trans Im. Proc.,2012), the contents of which are incorporated herein by reference, asfar as permitted.

As provided herein, the Phase 1 latent sharp image cost function and thePhase 2 latent sharp image cost function can be expressed more generallythan provided above. For example, the Phase 1 cost function of Equation(3) can be expressed more generally as follows:c(L)=Fidelity Term(s)+ω₁(∥N ₁(D _(x) *L)∥_(q) ^(q) +∥N ₁(D _(y) *L)∥_(q)^(q)).  Equation (16).Alternatively, the regularization term for the cost function of Equation16 can be expressed for a 1-norm gradient to be “∥N₁ grad L|∥₁ ¹.”

Further, the Phase 2 cost function of Equation (11) can be expressedmore generally as follows:c(L)=Fidelity Terms(s)+ω₂(∥N ₂(D _(x) *L)∥₂ ∥N ₂(D _(y) *L)∥₂ ²)+σ∥L−L_(Phase1)∥₂ ²   Equation (17).

The exact fidelity term(s) utilized in Equations (16) and (17) can vary.In one embodiment, the fidelity term(s) can be expressed as follows:Fidelity Terms(s)=α₀ ∥L _(j) *K−B _(j)∥₂ ²+α₁ |D _(x) *L _(j) *K−D _(x)*B _(j)∥₂ ² D _(y) *L _(j) *K−D _(y) *B _(j)∥₂ ²+α₁₁ ∥D _(xx) *L _(j)*K−D _(xx) *B _(j)∥₂ ²+α₁₂ ∥D _(xy) *L _(j) *K−D _(xy) *B _(j)∥₂ ²+α₂₂∥D _(yy) *L _(j) *K−D _(yy) *B _(j)∥₂ ²  Equation (18).In Equation (18), α₀, α₁, α₂, α₁₁, α₁₂, and _α₂₂ are fidelity termweights. It should be noted that the fidelity term weights can be chosento be zero (“0”) or another weight. As a result thereof, in Equation(18), this cost function covers cases both with and without derivatives.It should also be noted that the cost functions in Equations (16) and(17) can be minimized using variable splitting algorithm similar to thatprovided above. For example, the formulas can be pretty much the same asabove Equations (6, 7, 9 and 13, 14, 15), except the term α₁+α₂D wouldbe replaced everywhere by α₀+a, conj(F(D_(x)))F(D_(x))+α₂conj(F(D_(y)))F(D_(y))+α₁₁ conj(F(D_(xx)))F(D_(xx))+α₁₂conj(F(D_(xy)))F(D_(xy))+α₂₂ conj(F(D_(yy)))F(D_(yy)).

Deconvolution is very sensitive to any departures from a linear blurringmodel. Real-life images often contain highlights that have been clippeddue to sensor saturation. Clipping is strongly non-linear and violates atraditional linear blurring model. The highlights have high contrastedges that tend to cause heavy ringing artifacts in deblurred images.The multiple phase algorithm described above does not have thecapability to deal with burnt out highlights or other outliers. However,as provided herein, in certain embodiments of the present invention, afidelity term spatial mask M can be added to one or more of the fidelityterm(s) of the Phase 1 and Phase 2 cost functions to allow for specialtreatment of the outliers.

As provided herein, the Phase 1 cost function of Equation 16 can berewritten to include a fidelity term spatial mask M as follows:c(L)=α₁ ∥M(K*L−B)∥_(p) ^(p)+α₂(∥M(K*D _(x) *L−D _(x) *B)∥_(p) ^(p)+∥M(K*D _(y) *L−D _(y) *B)∥_(p) ^(p))+ω₁(∥N ₁(D _(x) *L)∥_(q) ^(q) +∥N₁(D _(y) *L)∥_(q) ^(q)  Equation (19)

Further, the Phase 2 cost function of Equation 17 can be rewritten toinclude a fidelity term spatial mask M as follows:c(L)=α₁ ∥M(K*L−B)∥_(p) ^(p)+α₂(∥M(K*D _(x) *L−D _(x) *B)∥_(p) ^(p)+∥M(K*D _(y) *L−D _(y) *B)∥_(p) ^(p))+ω₂(∥N ₂(D _(x) *L)∥_(q) ^(q) +∥N₂(D _(y) *L)∥_(q) ^(q))+σ₁ ∥L−L _(Phase1)∥_(q) ^(q)+σ₂(∥D _(x) *L−D _(x)*L _(Phase1)∥_(q) ^(q) +∥D _(y) *L−D _(y) *L _(Phase1)∥_(q)^(q)).  Equation (20)

In Equations 19 and 20, M is an adaptive fidelity spatial mask thatallows for special treatment of certain pixels, e.g. outliers. Forexample, the fidelity spatial mask M can be used to limit the areas ofthe image that are being processed to the regions that do not includeclipped highlights (e.g. limit processing to the areas far enough fromoutliers). With this design, the highlight regions can be deblurredseparately, and with different parameters from the regions that do notinclude clipped highlights.

FIG. 5A illustrates a multi-phase image deconvolution method with thecapability to deal with burnt out highlights or other outliers in ablurred image 500. It should be noted that the order of some of thesteps in FIG. 5A can be switched, performed concurrently, and/or some ofthe steps can be optional. In this embodiment, the multi-phase imagedeconvolution method can be considered a four phase non-blind imagedeconvolution method that includes special processing for outliers (e.g.highlights).

More specifically, in this embodiment, the four phase non-blind imagedeconvolution method can be divided into (i) Phase 1 processing 502; (i)Phase 2 processing 504; (iii) Phase 3 processing 506; and (iv) Phase 4processing 508. In this embodiment, (i) Phase 1 processing 502 issimilar to Phase 1 processing 202 described above and illustrated inFIG. 2, except the Phase 1 cost function includes a fidelity spatialmask M; (ii) Phase 2 processing 504 is similar to Phase 2 processing 204described above and illustrated in FIG. 2, except the Phase 2 costfunction includes a fidelity spatial mask M; (iii) Phase 3 processing506 is similar to Phase 1 processing 202 described above and illustratedin FIG. 2; and (iv) Phase 4 processing 508 is similar to Phase 2processing 204 described above and illustrated in FIG. 2.

With this design, (i) Phase 1 processing 502 of the four phase methodcan be used to restore the edges in the non-outlier regions; (ii) Phase2 processing 504 of the four phase method can be used to restore texturein the non-outlier regions; (iii) Phase 3 processing 506 of the fourphase method can be used to restore the edges in the outlier regions;and (iv) Phase 4 processing 508 of the four phase method can be used torestore texture in the outlier regions. Thus, in this embodiment, (i)Phase 1 processing 502 and Phase 2 processing 504 are used to restorethe non-outlier regions; and (ii) Phase 3 processing 506 and Phase 4processing 508 are used to restore the outlier regions.

Referring to FIG. 5A, at block 500, the blurred image is read by thealgorithm. Subsequently, at block 510, the blurred image is reviewed toidentify outlier pixels (regions) and non-outlier pixels (regions). Inone embodiment, the outlier regions are highlighted regions (regionswhere the image sensor was saturated), and the non-outlier regions arenon-highlighted regions. For example, pixels in the blurry image havinga brightness value over a certain brightness threshold can be labeled asoutlier pixels, and pixels having a brightness value under thebrightness threshold are labeled as non-outlier pixels. In certainembodiment, pixels in the neighborhood of the outlier pixels can also belabeled as outlier pixels.

After the outlier regions and non-outlier regions are identified,separate processing of these regions can begin. At block 512, of theadaptive fidelity spatial mask M is generated. As a non-exclusiveexample, the fidelity spatial mask M can be a binary mask having a valueof either 0 or 1. In this example, the fidelity spatial mask M₁ can havea value of 0 for pixels that are labeled as outlier pixels, and a valueof 1 for pixels that are labeled as non-outlier pixels. In this example,(i) the outliers and its neighboring pixels will be assigned the valueof 0; and (ii) the remaining pixels will be assigned the value of 1.

Alternatively, the pixels of the fidelity spatial mask M can have valuessomewhere in the range of 0 to 1. In this example, (i) the outliers andits neighboring pixels can be assigned the value of 0; (ii) the pixelsfar enough away from the outliers will be assigned the value of 1; and(iii) the remaining pixels will have somewhere between zero and onedepending on how far they are from an outlier. This will lead to moresmooth transitions. FIG. 5B is a simplified illustration of a portion ofa fidelity spatial mask M 550.

Referring back to FIG. 5A, subsequently, at block 502, Phase 1processing is performed to restore the main edges in the non-outlierregions with a strong emphasis on suppressing ringing. Phase 1processing 502 here is similar to the Phase 1 processing 202 describedabove in reference to FIG. 2, except the phase 1 cost function includesthe fidelity spatial mask M in the fidelity term(s), in addition to theadaptive regularization spatial mask N₁ and Laplacian or Hyper-Laplacianprior regularization. In this embodiment, the fidelity spatial mask M isused to limit deconvolution to the areas far enough from outliers, whichsuppresses ringing around outliers. Similar to the embodiment describedin FIG. 2, Phase 1 processing 502 can include Subphases 1A, 1B, 1C, andetc. with each subsequent Subphase having decreased regularization andan improved adaptive regularization spatial mask.

After Phase 1 processing of the non-outlier regions is complete, Phase 2processing 504 of the non-outlier regions is performed to restoretexture at block 504. Phase 2 processing 504 here is similar to thePhase 2 processing described above in reference to FIG. 2, except thephase 2 cost function includes the fidelity spatial mask M in thefidelity term(s), in addition to the adaptive regularization spatialmask N₂, the Gaussian prior regularization, and the spatial priorregularization. In Phase 2 processing 504, the fidelity spatial mask Mis used in the fidelity term(s) of the cost function to limitdeconvolution to the areas far enough from outliers. This can be used tosuppress ringing around outliers, and the outlier areas can get strongerregularization.

As result thereof, the Phase 2 latent sharp image contains relativelyfew ringing artifacts and noise, and more fine texture. The Phase 2latent sharp image generated from Phase 2 processing 504 of thenon-outlier regions is used to generate the final latent sharp image516.

Next, at block 506, Phase 3 processing is performed to restore the mainedges in the outlier regions with a strong emphasis on suppressingringing. Phase 3 processing 506 here is similar to the Phase 1processing 202 described above in reference to FIG. 2, except (i)stronger regularization is used to suppress ringing; and (ii) thefidelity spatial mask M used for Phase 3 processing (“Phase 3 fidelityspatial mask M₃”) needs to be changed so the outlier regions getrestored. For example, the Phase 3 processing cost function could either(i) skip the fidelity spatial mask M₃ entirely and restore the entireimage, or (ii) use one minus the Phase 1 fidelity spatial mask M as thePhase 3 fidelity spatial mask M₃ (M₃=1−M) to limit restoration to theoutlier regions. Similar to the embodiment described in FIG. 2, Phase 3processing 506 can include Subphases 3A, 3B, 3C, and etc. with eachsubsequent Subphase having decreased regularization and an improvedadaptive regularization spatial mask.

After Phase 3 processing of the outlier regions is complete, Phase 4processing is performed to restore texture to the outlier regions atblock 508 and generate a Phase 4 latent sharp image. Phase 4 processing508 here is similar to the Phase 2 processing described above inreference to FIG. 2, except (i) stronger regularization is used tosuppress ringing; and (ii) the fidelity spatial mask M used for Phase 4processing (“Phase 4 fidelity spatial mask M₄”) needs to be changed sothe outlier regions get restored. For example, the Phase 4 processingcost function could either (i) skip the fidelity spatial mask M₄entirely and restore the entire image, or (ii) use one minus the Phase 2fidelity spatial mask M as the Phase 4 fidelity spatial mask M₄ (M₄=1−M)to limit restoration to the outlier regions.

Next at block 514, the Phase 2 latent sharp image is merged with thePhase 4 latent sharp image to create the output latent sharp image 516.For example, the outlier areas of the Phase 4 latent sharp image can becombined with the non-outlier areas of the Phase 2 latent sharp image togenerate the output latent sharp image 516. With this design, themulti-phase method provided herein can be used to separately deblur thehighlight regions, with different parameters from the non-highlightregions.

In one embodiment, the final output (latent sharp image L) can beexpressed as follows:L=M _(Blend) L _(Phase2)+(1−M _(Blend))L _(Phase4).  Equation (21)In Equation (21) (i) M_(Blend) is a blending mask, (ii) L_(Phase2) isthe latent sharp image after Phase 2 processing, and (iii) L_(Phase4) isthe latent sharp image after Phase 4 processing. In one embodiment, theblending mask M_(Blend) M is the same fidelity spatial mask M used inPhase 1 processing 502 and Phase 2 processing 504. With this design, ifthe mask M is not binary, pixels in areas far enough from outliers areassigned the result of Phase 2 processing 504, and pixels in outlierareas are assigned the result of Phase 4 processing 508, and there issmooth blending in between.

It should be noted that in this embodiment, when there are no outliersin the blurry image (no outliers identified in block 510), Phase 3 and 4processing 506, 508 can be skipped, and the final output latent sharpimage 516 is the Phase 2 latent sharp image.

Further, the user can also choose to (i) skip Phase 3 and 4 processing506, 508, and (ii) use the Phase 2 latent sharp image as the finaloutput latent sharp image 516, independently of whether or not anyoutliers are detected. For example, if the PSF is small, the area aroundthe outliers that is left blurry is relatively small and the blur may beless annoying than the ringing artifacts introduced by Phase 3 and 4processing 506, 508. Further, skipping Phase 3 and 4 processing 506, 508improves the speed of the algorithm.

One non-exclusive example of a suitable Phase 1 cost function for Phase1 processing 502 of the non-outlier regions is provided below:c(L)=α₁ ∥M K*L−B)∥₂ ²+α₂(∥M(K*D _(x) *L−D _(x) *B)∥₂ ² +∥M(K*D _(y) *L−D_(y) *B)∥₂ ²)+ω₁(∥N ₁(D _(x) *L)∥₁ ¹ +∥N ₁ ¹(D _(y) *L)∥₁ ¹)  Equation(22)

In one non-exclusive example, the fidelity weight parameter α₁ can beset to be equal to 0.001 (α₁=0.001), and the fidelity weight parameterα₂ can be set to be equal to 1 (α₂=1). As a result thereof, the costfunction relies mainly on fidelity terms with derivatives which help toreduce ringing. In Equation (22), M is an adaptive fidelity spatialmask, N₁ is an adaptive regularization spatial mask, and ω₁ is aregularization weight in the smooth areas.

The cost function of Equation (22) can be minimized by variablesplitting and alternating minimization. In certain instances,approximately 6-8 iterations are needed to suppress ringing. The costfunction of Equation (22) after splitting can be expressed as follows:c(L,R,W _(x) ,W _(y))=α₁(∥MR−B)∥₂ ² +γ∥K*L−R∥ ₂ ²)+α₂(∥M(D _(x) *R−D_(x) *B)∥₂ ² +∥M(D _(y) *R−D _(y) *B)∥₂ ²+γ(∥D _(x) *K*L−D _(x) *R∥ ₂ ²+∥D _(y) *K*L−D _(y) *R∥ ₂ ²))+ω₁(∥N ₁ W _(x)∥₁ ¹ +∥N ₁ W _(y)∥₁ ¹+(∥W_(y) −D _(x) *L∥ ₂ ² +∥W _(y) D _(y) *L)∥₂ ²)).  Equation (23).

As a non-exclusive example, penalty weights γ=β=1 can be used in thefirst iteration, and increased by the factor √{square root over (2)}(for γ), and respectively 2 (for β) in each iteration.

The minimization formulas are as follows:

$\begin{matrix}{R = \frac{{MB} + {\gamma\; K*L}}{M + \gamma}} & {{Equation}\mspace{14mu}(24)} \\{{W_{x} = {{sthr}\left( {{D_{x}^{*}L},\frac{N}{2\beta}} \right)}},} & {{Equation}\mspace{14mu}(25)} \\{W_{y} = {{{sthr}\left( {{D_{y}^{*}L},\frac{N}{2\beta}} \right)}.}} & {{Equation}\mspace{14mu}(26)}\end{matrix}$

Further, there is a closed form formula for the minimum of the latentsharp image cost function of Equation (23) as follows:

$\begin{matrix}{{F(L)} = {\frac{\begin{matrix}{{\left( {{\alpha 1} + {{\alpha 2}\; D}} \right)\overset{\_}{F(K)}{F(R)}} +} \\{\frac{\omega_{1}\beta}{\gamma}\left( {{\overset{\_}{F\left( D_{x} \right)}{F\left( W_{x} \right)}} + {\overset{\_}{F\left( D_{y} \right)}{F\left( W_{y} \right)}}} \right)}\end{matrix}}{{\left( {{\alpha 1} + {{\alpha 2}\; D}} \right)\overset{\_}{F(K)}{F(K)}} + {\frac{\omega_{1}\beta}{\gamma}D}}.}} & {{Equation}\mspace{14mu}(27)}\end{matrix}$In Equation (26), D has the same value as in Equation (10).

One non-exclusive example of a suitable Phase 2 cost function for Phase2 processing 504 of the non-outlier regions is provided below:c(L)=α₁ ∥M(K*L−B)∥₂ ²+α₂(∥M(K*D _(x) *L−D _(x) *B)∥₂ ² +∥M(K*D _(y) *L−D_(y) *B)∥₂ ²)+ω₂(∥N ₂(D _(x) *L)∥₂ ²+∥(N ₂(D _(x) *L,∥ ₂ ² +∥N ₂(D _(y)*L)∥₂ ²+σ₁ ∥L−L _(Phase1)∥₂ ²+σ₂(∥D _(x) *L−D _(x) *L _(Phase1)∥₂ ² +∥D_(y) *L−D _(y) *L _(Phase1)∥₂ ²).  Equation (28)

The Phase 2 cost function of Equation (28) can be minimized by variablesplitting and alternating minimization. In certain instances, no morethan approximately 3 iterations seem to be needed. The cost function ofEquation (28) can be expressed after splitting as follows:c(L,R,W _(x) ,W _(y))=α₁(∥M(R−B)∥₂ ² +γ∥K*L−R∥ ₂ ²)+α₂(∥M(D _(x) *R−D_(x) *B)∥₂ ² +∥M(D _(y) *R−D _(y) *B)∥₂ ²+γ(∥D _(x) *K*L−D _(x) *R∥ ₂ ²+∥D _(y) *K*L−D _(y) *R∥ ₂ ²))+ω₂(∥N ₂ W _(x)∥₁ ¹ +∥N ₂ W _(y)∥₁ ¹+β(W_(x) −D _(x) *L∥ ₂ ² +∥W _(y) −D _(y) *L∥ ₂ ²))+σ₁ ∥L−L _(Phase1)∥₂²+σ₂(∥D _(x) *L _(Phase1)∥₂ ² +∥D _(y) *L−D _(y) *L−D _(y) *L_(Phase1)∥₂ ²).  Equation (29)

In one, non-exclusive embodiment, penalty weights γ=β=1 are used infirst iteration, and increased by the factor √{square root over (2)}(for α) and respectively 2 (for β) in each iteration.

The following minimization formulas can be utilized:

$\begin{matrix}{R = \frac{{MB} + {\gamma\; K*L}}{M + \gamma}} & {{Equation}\mspace{14mu}(30)} \\{{W_{x} = \frac{D_{x}^{*}L}{1 + {\frac{1}{\beta}N\; 2^{2}}}},} & {{Equation}\mspace{14mu}(31)} \\{W_{y} = \frac{D_{y}^{*}L}{1 + {\frac{1}{\beta}N\; 2^{2}}}} & {{Equation}\mspace{14mu}(32)}\end{matrix}$

Further, there is a closed form formula for the minimum of the latentsharp image cost function of Equation (29) as follows:

$\begin{matrix}{{F(L)} = {\frac{\begin{matrix}{{\left( {{\alpha 1} + {{\alpha 2}\; D}} \right)\overset{\_}{F(K)}{F(R)}} +} \\\begin{matrix}{{\left( {\sigma_{1} + {\sigma_{2}D}} \right){F\left( L_{{Phase}\; 1} \right)}} +} \\{\omega_{2}\frac{\beta}{\gamma}\left( {{\overset{\_}{F\left( D_{x} \right)}{F\left( W_{x} \right)}} + {\overset{\_}{F\left( D_{y} \right)}{F\left( W_{y} \right)}}} \right)}\end{matrix}\end{matrix}}{{\left( {{\alpha 1} + {{\alpha 2}\; D}} \right)\overset{\_}{F(K)}} + \left( {\sigma_{1} + {\sigma_{2}D}} \right) + {\omega_{2}\beta_{\gamma}D}}.}} & {{Equation}\mspace{14mu}(33)}\end{matrix}$In Equation (33), D has the same value as in Equation (10).

One non-exclusive example of a suitable Phase 3 cost function for Phase3 processing 506 of the outlier regions is provided below:c(L)=α₁∥(K*L−B)∥₂ ²+α₂(∥(K*D _(x) *L−D _(x) *B)∥₂ ²+∥(K*D _(y) *L−D _(y)*B)∥₂ ²)+ω₁(∥N ₁(D _(x) *L)∥₁ +∥N ₁(D _(y) *L)∥₁).  Equation (34)It should be noted that Equation (34) is the same as Equation (4)provided above. In this Phase 3 cost function, no fidelity spatial maskM₃ is used. The Phase 3 cost function of Equation (34) can be minimizediteratively, and one possibility includes variable splitting as providedabove in Equations 6, 7, and 9 with alternating minimization. It shouldalso be noted that the adaptive regularization spatial mask N₁ can becalculated using the Phase 2 latent sharp image. It should also be notedthat the restoration in highlight areas uses more regularization and isthus more crude.

In Phase 4 processing 508, some texture is resorted in and around theoutliers. One non-exclusive example of a suitable Phase 4 cost functionfor Phase 4 processing 508 of the outlier regions is provided below:

Given blurry image B and PSF kernel K, the Phase 4 cost function can beexpressed as follows:c(L)=α₁ ∥K*L−B∥ ₂ ²+α₂(∥K*D _(x) *L−D _(x) *B∥ ₂ ² +∥K*D _(y) *L−D _(y)*B∥ ₂ ²)+ω₂(∥N ₂(D _(x) *L)∥₂ ² +∥N ₂(D _(y) *L)∥₂ ²)+σ₁ ∥L−L_(Phase3)∥₂ ²+σ₂(∥D _(x) *L−D _(x) *L _(Phase3)∥₂ ² +∥D _(y) *L−D _(y)*L _(Phase3)∥₂ ²).  Equation (35)It should be noted that Equation (35) is somewhat similar to Equation(11) provided above. In this Phase 4 cost function, no fidelity spatialmask M₄ is used. The Phase 4 cost function of Equation (35) can beminimized iteratively, and one possibility includes variable splittingsomewhat similar to Equations 13-15 with alternating minimization. Inone embodiment, there is a closed form formula for the minimum asfollows:

$\begin{matrix}{{F(L)} = {\frac{\begin{matrix}{{\left( {{\alpha 1} + {{\alpha 2}\; D}} \right)\overset{\_}{F(K)}{F(B)}} +} \\\begin{matrix}{{\left( {\sigma_{1} + {\sigma_{2}D}} \right){F\left( L_{{Phase}\; 3} \right)}} +} \\{\omega_{2}{\beta\left( {{\overset{\_}{F\left( D_{x} \right)}{F\left( W_{x} \right)}} + {\overset{\_}{F\left( D_{y} \right)}{F\left( W_{y} \right)}}} \right)}}\end{matrix}\end{matrix}}{{\left( {{\alpha 1} + {{\alpha 2}\; D}} \right)\overset{\_}{F(K)}{F(K)}} + \left( {\sigma_{1} + {\sigma_{2}D}} \right) + {\omega_{2}\beta\; D}}.}} & {{Equation}\mspace{14mu}(36)}\end{matrix}$

It should be noted that Phase 4 cost function of Equation 35 containsthe option to post-process the latent sharp image in each iteration, bylimiting the difference between L and L_(Phase3). The idea was to reduceringing, but, in certain instances, it causes bad posterization.Therefore this option can be switched off by default.

FIG. 6A is a simplified view of a blurry image 600 that includeshighlighted (outlier) regions 610 (illustrated in white), andnon-highlighted (non-outlier) regions 612. FIG. 6B is a simplified viewof a Phase 2 latent sharp image 602. At the completion of Phase 2processing, no outlier region restoration has occurred.

FIG. 6C is a simplified view of the final latent sharp image 604 thatwas generated by combining the Phase 2 latent sharp image 602(illustrated in FIG. 6B) with the Phase 4 latent sharp image. The finallatent sharp image 604 includes outlier region restoration.

With reference to FIGS. 6B and 6C, the results of Phase 2 processingcontain very little ringing, but the outlier shape is not properlyrestored and the areas around outliers remain blurry. Phase 4 processinghelps to restore outlier shape and some texture around outliers.

In yet another embodiment, the present invention is directed to amultiple phase, non-blind deconvolution method with outlier processingin the Luminance-Chrominance Color Space. As provided herein, deblurredimages commonly suffer from strong ringing artifacts that are caused bydepartures from the linear image formation and blurring model,inaccuracy of the PSF, and other factors. Thus, the ringing artifactsare extremely hard to avoid. Further, as provided herein, whendeblurring is applied separately to each color channel of RGB (red,blue, green color space) image, the ringing artifacts can have a strongcolor. Thus, color artifacts can be more conspicuous and annoying.

In one embodiment, the present invention reduces the color artifacts byperforming deblurring in some color space that separates luminance(brightness component) from chrominance (color components). In thisembodiment, because, the human visual system is less sensitive tochrominance sharpness, the regularization can be increased for thechrominance channels, to reduce color ringing. Further, the deblurredluminance channel can be used for generating the adaptive regularizationmasks for chrominance channels.

Moreover, in certain embodiments, the number of phases can be reducedfor the chrominance channels. This will speed up the algorithm.

FIG. 7 is an algorithm flowchart for a multi-phase, non-blinddeconvolution method in performed primarily in the YCbCr color space. Atstep 700, the blurred image is inputted into the system, the raw imagedata is read, the raw image data is demosaiced, and the image isconverted to a color space that includes luminance and chrominancechannels (e.g. the YCbCr color space). In one embodiment, themulti-phase approach is the 4-phase approach described above andillustrated in FIG. 5 is performed on the luminance channel 701L andsubsequently on the two chrominance channels 701C. It should be notedthat either the Cb chrominance channel or the Cr chrominance channel canbe referred to as a first or second chrominance channel.

More specially, the blurred image is inputted into block 701L, andprocessing of the luminance channel begins. At block 710, the blurredimage is reviewed to identify outlier pixels (regions) and non-outlierpixels (regions) (similar to block 510). At block 712L, the adaptivefidelity spatial mask M is generated (similar to block 512).

At block 702L, Phase 1 processing is performed on the luminance channelto restore the main edges in the non-outlier regions with a strongemphasis on suppressing ringing (similar to block 502). Phase 1processing 702L includes the phase 1 cost function the includes thefidelity spatial mask M in the fidelity term(s), in addition to theadaptive regularization spatial mask N₁ and Laplacian or Hyper-Laplacianprior regularization. Phase 1 processing 702L can include Subphases 1A,1B, 1C, and etc. with each subsequent Subphase having decreasedregularization and an improved adaptive regularization spatial mask.

Next, Phase 2 processing of the non-outlier regions on the luminancechannel is performed to restore texture at block 704L (similar to block504) and generate the Phase 2 luminance latent sharp image (L_(Ph2) _(_)_(Lum)) Phase 2 processing includes the phase 2 cost function having thefidelity spatial mask M in the fidelity term(s), in addition to theadaptive regularization spatial mask N₂, the Gaussian priorregularization, and the spatial prior regularization

Subsequently, at block 706L, Phase 3 processing on the luminance channelis performed to restore the main edges in the outlier regions with astrong emphasis on suppressing ringing (similar to block 506). Phase 3processing 706L can include Subphases 3A, 3B, 3C, and etc. with eachsubsequent Subphase having decreased regularization and an improvedadaptive regularization spatial mask.

Next, at block 708L, Phase 4 processing on the luminance channel isperformed to restore texture to the outlier regions and generate a Phase4 luminance latent sharp image (L_(Ph4) _(_) _(Lum)) (similar to block508).

Next at block 714L, and with reference to FIG. 8A, the Phase 2 luminancelatent sharp image (L_(Ph2) _(_) _(Lum)) 810 is merged with the Phase 4luminance latent sharp image (L_(Ph4) _(_) _(Lum)) 812 to create theluminance channel latent sharp image (L_(Lum)) 814. For example, theoutlier areas of the Phase 4 latent sharp image can be combined with thenon-outlier areas of the Phase 2 latent sharp image to generate theluminance channel latent sharp image (L_(Lum)) 814.

Referring back to FIG. 7, at block 701C, processing 701C of the twochrominance channels begins. At block 715, of the adaptiveregularization spatial mask N₁ is updated/generated using the luminancechannel latent sharp image (L_(Lum)) 814.

At block 702C, Phase 1 processing is performed on each chrominancechannel to restore the main edges in the non-outlier regions with astrong emphasis on suppressing ringing (similar to block 502). For eachchrominance channel, Phase 1 processing 702L can include using the phase1 cost function that includes the fidelity spatial mask M in thefidelity term(s), in addition to the adaptive regularization spatialmask N₁ and Laplacian or Hyper-Laplacian prior regularization. For eachchrominance channel, Phase 1 processing 702C can include Subphases 1A,1B, 1C, and etc. with each subsequent Subphase having decreasedregularization and an improved adaptive regularization spatial mask.

Next, Phase 2 processing of the non-outlier regions is performed on thetwo chrominance channels to restore texture at block 704C (similar toblock 504), and generate (i) a Phase 2 Cb chrominance latent sharp image(L_(Ph2) _(_) _(CbChr)), and (ii) a Phase 2 Cr chrominance latent sharpimage (L_(Ph2) _(_) _(CrChr)) For each chrominance channel, Phase 2processing can include using the phase 2 cost function having thefidelity spatial mask M in the fidelity term(s), in addition to theadaptive regularization spatial mask N₁, the Gaussian priorregularization, and the spatial prior regularization.

Subsequently, at block 706C, Phase 3 processing is performed on the twochrominance channels to restore the main edges in the outlier regionswith a strong emphasis on suppressing ringing (similar to block 506).For each chrominance channel, Phase 3 processing 706C can includeSubphases 3A, 3B, 3C, and etc. with each subsequent Subphase havingdecreased regularization and an improved adaptive regularization spatialmask.

Next, at block 708C, Phase 4 processing (similar to block 508) isperformed on each chrominance channel, to restore texture to the outlierregions, and generate (i) a Phase 4 Cb chrominance latent sharp image(L_(Ph4) _(_) _(CbChr)), and (ii) a Phase 4 Cr chrominance latent sharpimage (L_(Ph4) _(_) _(CrChr))

Next at block 714C, (i) with reference to FIG. 8B, the Phase 2 Cbchrominance latent sharp image (L_(Ph2) _(_) _(CbChr)) 816 is mergedwith the Phase 4 Cb chrominance latent sharp image (L_(Ph4) _(_)_(CbChr)) 818 to create a Cb chrominance channel latent sharp image(L_(CbChr)) 820; and (ii) with reference to FIG. 8C, the Phase 2 Crchrominance latent sharp image (L_(Ph2) _(_) _(CrChr)) 822 is mergedwith the Phase 4 Cr chrominance latent sharp image (L_(Ph4) _(_)_(CrChr)) 824 to create a Cr chrominance channel latent sharp image(L_(CrChr)) 826. For example, the non-outlier areas of the Phase 2 Cbchrominance latent sharp image (L_(Ph2) _(_) _(CbChr)) 816 can becombined with the outlier areas of the Phase 4 Cb chrominance latentsharp image (L_(Ph2) _(_) _(CbChr)) 818 to create the Cb chrominancechannel latent sharp image (L_(CbChr)) 820. Similarly, the non-outlierareas of the Phase 2 Cr chrominance latent sharp image (L_(Ph2) _(_)_(CrChr)) 822 can be combined with the outlier areas of the Phase 4 Crchrominance latent sharp image (L_(Ph4) _(_) _(CrChr)) 824 to create theCr chrominance channel latent sharp image (L_(CrChr)) 826.

Subsequently, at block 720 final processing occurs. In this block, withreference to FIG. 8D, the luminance channel latent sharp image (L_(Lum))814, the Cb chrominance channel latent sharp image (L_(CbChr)) 820, andthe Cr chrominance channel latent sharp image (L_(CrChr)) 826 can beconverted to the RGB color space, gamma and contrast curves can beapplied, and other operations (e.g. sharpening) is performed to generatethe final latent sharp image (L_(Final)) 828. Finally, at block 716 thefinal latent sharp image (L_(Final)) 828 is output.

While the current invention is disclosed in detail herein, it is to beunderstood that it is merely illustrative of the presently preferredembodiments of the invention and that no limitations are intended to thedetails of construction or design herein shown other than as describedin the appended claims.

What is claimed is:
 1. A method for deblurring a blurry image thatincludes main edges and texture, the method comprising the steps of:identifying any outlier regions in the blurry image; creating a fidelityterm mask based on the identified outlier regions; performing a firstphase of deconvolution with a first phase regularization spatial mask toreconstruct the main edges and generate a first phase latent sharp imagehaving reconstructed main edges; wherein the first phase ofdeconvolution includes using a first phase algorithm that uses thefidelity term mask; and performing a second phase of deconvolution witha second phase regularization spatial mask to reconstruct the textureand generate a second phase latent sharp image, the second phaseregularization spatial mask being different from the first phaseregularization spatial mask; wherein the step of performing a secondphase of deconvolution includes using a second phase algorithm that usesthe fidelity term mask.
 2. The method of claim 1 further comprising thestep of generating the second phase regularization spatial mask usingthe first phase latent sharp image.
 3. The method of claim 1 wherein (i)the step of performing a first phase includes using a first phasealgorithm that assumes a Laplacian or Hyper-Laplacian prior and includesthe first phase regularization spatial mask to suppress ringingartifacts and noise, and (ii) the step of performing a second phaseincludes using a second phase algorithm that assumes a Gaussian prior,and includes a spatial prior, and the second phase regularizationspatial mask to help to recover image texture without introducing strongringing artifacts.
 4. The method of claim 1 wherein (i) the step ofperforming a first phase includes using the first phase algorithm thatassumes a Laplacian or Hyper-Laplacian prior and includes the firstphase regularization spatial mask to suppress ringing artifacts andnoise, and (ii) the step of performing a second phase includes using thesecond phase algorithm that assumes a Gaussian prior, a spatial prior,and includes the second phase regularization spatial mask to help torecover image texture without introducing strong ringing artifacts. 5.The method of claim 1 further comprising the steps of (i) performing athird phase of deconvolution on the blurry image with a third phaseregularization spatial mask to reconstruct the main edges in at leastone of the outlier regions to generate a third phase latent sharp image;(ii) performing a fourth phase of deconvolution with a fourth phaseregularization spatial mask to reconstruct the texture in at least oneof the outlier regions to generate a fourth phase latent sharp image,the fourth phase regularization spatial mask being different from thethird phase regularization spatial mask; and (iii) identifying anynon-outlier regions in the blurry image; wherein the first phase ofdeconvolution reconstructs the main edges in at least one of thenon-outlier regions; and wherein the second phase of deconvolutionreconstructs the texture in at least one of the non-outlier regions. 6.The method of claim 5 further comprising the step of merging the secondphase latent sharp image with the fourth phase latent sharp image tocreate the output latent sharp image.
 7. The method of claim 5 wherein(i) the step of performing a first phase includes using the first phasealgorithm that assumes a Laplacian or Hyper-Laplacian prior and includesthe first phase regularization spatial mask to suppress ringingartifacts and noise, (ii) the step of performing a second phase includesusing the second phase algorithm that assumes a Gaussian prior, andincludes a spatial prior, and the second phase regularization spatialmask to help to recover image texture without introducing strong ringingartifacts; (iii) the step of performing a third phase includes using athird phase algorithm that assumes a Laplacian or Hyper-Laplacian priorand includes the third phase regularization spatial mask to suppressringing artifacts and noise, and (iv) the step of performing a fourthphase includes using a fourth phase algorithm that assumes a Gaussianprior, and includes a spatial prior, and the fourth phase regularizationspatial mask to help to recover image texture without introducing strongringing artifacts.
 8. A system for deblurring a blurry image thatincludes main edges and texture, the system comprising: a control systemthat includes a processor that (i) identifies any outlier regions in theblurry image; (ii) creates a fidelity term mask based on the identifiedoutlier regions; (iii) performs a first phase of deconvolution with afirst phase regularization spatial mask to reconstruct the main edgesand generate a first phase latent sharp image having reconstructed mainedges; wherein the first phase of deconvolution uses a first phasealgorithm that uses the fidelity term mask; and iv performs a secondphase of deconvolution with a second phase regularization spatial maskto reconstruct the texture and generate a second phase latent sharpimage, the second phase regularization spatial mask being different fromthe first phase regularization spatial mask; wherein the second phase ofdeconvolution uses a second phase algorithm that uses the fidelity termmask.
 9. The system of claim 8 wherein the control system generates thesecond phase regularization spatial mask using the first phase latentsharp image.
 10. The system of claim 8 wherein (i) the first phase ofdeconvolution uses a first phase algorithm that assumes a Laplacian orHyper-Laplacian prior and includes the first phase regularizationspatial mask to suppress ringing artifacts and noise, and (ii) thesecond phase of deconvolution uses a second phase algorithm that assumesa Gaussian prior, and includes a spatial prior, and the second phaseregularization spatial mask to help to recover image texture withoutintroducing strong ringing artifacts.
 11. The system of claim 8 wherein(i) the first phase algorithm assumes a Laplacian or Hyper-Laplacianprior and includes the first phase regularization spatial mask tosuppress ringing artifacts and noise, and (ii) the second phasealgorithm assumes a Gaussian prior, and includes a spatial prior, andthe second phase regularization spatial mask to help to recover imagetexture without introducing strong ringing artifacts.
 12. The system ofclaim 8 wherein the control system (i) performs a third phase ofdeconvolution on the blurry image with a third phase regularizationspatial mask to reconstruct the main edges in at least one of theoutlier regions to generate a third phase latent sharp image; and (ii)performs a fourth phase of deconvolution with a fourth phaseregularization spatial mask to reconstruct the texture in at least oneof the outlier regions to generate a fourth phase latent sharp image,the fourth phase regularization spatial mask being different from thethird phase regularization spatial mask; and (iii) identifying anynon-outlier regions in the blurry image; wherein the first phase ofdeconvolution reconstructs the main edges in at least one of thenon-outlier regions; and wherein the second phase of deconvolutionreconstructs the texture in at least one of the non-outlier regions. 13.The system of claim 12 wherein the control system merges the secondphase latent sharp image with the fourth phase latent sharp image tocreate the output latent sharp image.
 14. The system of claim 12 wherein(i) the first phase algorithm assumes a Laplacian or Hyper-Laplacianprior and includes the first phase regularization spatial mask tosuppress ringing artifacts and noise, (ii) the second phase algorithmassumes a Gaussian prior, and includes a spatial prior, and the secondphase regularization spatial mask to help to recover image texturewithout introducing strong ringing artifacts; (iii) the third phase ofregularization includes using a third phase algorithm that assumes aLaplacian or Hyper-Laplacian prior and includes the third phaseregularization spatial mask to suppress ringing artifacts and noise, and(iv) the fourth phase of regularization includes using a fourth phasealgorithm that assumes a Gaussian prior, and includes a spatial prior,and the fourth phase regularization spatial mask to help to recoverimage texture without introducing strong ringing artifacts.
 15. Thesystem of claim 8 further comprising a capturing system that captures animage of the scene.
 16. A method for deblurring a blurry image thatincludes main edges and texture, the method comprising the steps of:converting the blurry image to a color space that includes a luminancechannel and a chrominance channel; performing a first phase ofdeconvolution on the luminance channel of the blurry image with a firstphase regularization spatial mask to reconstruct the main edges of theluminance channel and generate a luminance channel, first phase latentsharp image having reconstructed main edges; performing a second phaseof deconvolution on the luminance channel, first phase latent sharpimage with a second phase regularization spatial mask to reconstruct thetexture and generate a luminance channel, second phase latent sharpimage, the second phase regularization spatial mask being different fromthe first phase regularization spatial mask; wherein the second phaseregularization spatial mask is generated using the luminance channel,first phase latent sharp image; performing a first phase ofdeconvolution on the chrominance channel of the blurry image toreconstruct the main edges of the chrominance channel; and performing asecond phase of deconvolution on the chrominance channel to reconstructthe texture in edges of the chrominance channel.
 17. A system fordeblurring a blurry image that includes main edges and texture, thesystem comprising: a control system that includes a processor that (i)converts the blurry image to a color space that includes a luminancechannel and chrominance channels, (ii) performs a first phase ofdeconvolution on the luminance channel of the blurry image with a firstphase regularization spatial mask to reconstruct the main edges of theluminance channel and generate a luminance channel, first phase latentsharp image having reconstructed main edges; performs a second phase ofdeconvolution on the luminance channel, first phase latent sharp imagewith a second phase regularization spatial mask to reconstruct thetexture and generate a luminance channel, second phase latent sharpimage, the second phase regularization spatial mask being different fromthe first phase regularization spatial mask; wherein the second phaseregularization spatial mask is generated using the luminance channel,first phase latent sharp image; (iii) performs a first phase ofdeconvolution on the chrominance channel of the blurry image toreconstruct the main edges of the chrominance channel; and (iv) performsa second phase of deconvolution on the chrominance channel toreconstruct the texture in edges of the chrominance channel.
 18. Amethod for deblurring a blurry image with a known or estimated pointspread function, the method comprising the steps of: identifying anyoutlier regions in the blurry image; creating a fidelity term mask basedon the identified outlier regions; performing a first phase ofdeconvolution with a first latent sharp image cost function that assumesa Laplacian or Hyper-Laplacian prior regularization to generate a firstphase latent sharp image having reconstructed main edges, wherein thefirst latent sharp image cost function uses the fidelity term mask; andperforming a second phase of deconvolution with a second latent sharpimage cost function that assumes a Gaussian prior regularization andincludes a spatial prior regularization to generate a second phaselatent sharp image with reconstructed texture, wherein the second latentsharp image cost function uses the fidelity term mask.
 19. The method ofclaim 18 wherein the step of performing a first phase of deconvolutionincludes using a first phase regularization spatial mask in the firstlatent sharp image cost function; wherein the step of performing asecond phase of deconvolution includes using a second phaseregularization spatial mask in the second latent sharp image costfunction; and wherein the method includes generating the second phaseregularization spatial mask using the first phase latent sharp image.20. The method of claim 18 further comprising the steps of (i)performing a third phase of deconvolution on the blurry image with athird phase regularization spatial mask to reconstruct the main edges inat least one of the outlier regions to generate a third phase latentsharp image; (ii) performing a fourth phase of deconvolution with afourth phase regularization spatial mask to reconstruct the texture inat least one of the outlier regions to generate a fourth phase latentsharp image, the fourth phase regularization spatial mask beingdifferent from the third phase regularization spatial mask; and (iii)identifying any non-outlier regions in the blurry image; wherein thefirst phase of deconvolution reconstructs the main edges in at least oneof the non-outlier regions; and wherein the second phase ofdeconvolution reconstructs the texture in at least one of thenon-outlier regions.
 21. The method of claim 20 further comprising thestep of merging the second phase latent sharp image with the fourthphase latent sharp image to create the output latent sharp image. 22.The method of claim 20 wherein (i) the step of performing a third phaseincludes using a third phase algorithm that assumes a Laplacian orHyper-Laplacian prior and includes the third phase regularizationspatial mask to suppress ringing artifacts and noise, and (ii) the stepof performing a fourth phase includes using a fourth phase algorithmthat assumes a Gaussian prior, and includes a spatial prior, and thefourth phase regularization spatial mask to help to recover imagetexture without introducing strong ringing artifacts.
 23. A system fordeblurring a blurry image that includes main edges and texture, thesystem comprising: a control system that includes a processor that (i)identifies any outlier regions in the blurry image; (ii) creates afidelity term mask based on the identified outlier regions; (iii)performs a first phase of deconvolution with a first latent sharp imagecost function that assumes a Laplacian or Hyper-Laplacian priorregularization to generate a first phase latent sharp image havingreconstructed main edges; wherein the first phase of deconvolution usesa first phase algorithm that uses the fidelity term mask; and ivperforms a second phase of deconvolution with a second latent sharpimage cost function that assumes Gaussian prior regularization andincludes a spatial prior regularization to generate a second phaselatent sharp image with reconstructed texture; wherein the second phaseof deconvolution uses a second phase algorithm that uses the fidelityterm mask.
 24. The system of claim 23 wherein the first phase ofdeconvolution includes using a first phase regularization spatial maskin the first latent sharp image cost function; wherein the second phaseof deconvolution includes using a second phase regularization spatialmask in the second latent sharp image cost function; and wherein themethod includes generating the second phase regularization spatial maskusing the first phase latent sharp image.
 25. The system of claim 23wherein the control system (i) performs a third phase of deconvolutionon the blurry image with a third phase regularization spatial mask toreconstruct the main edges in at least one of the outlier regions togenerate a third phase latent sharp image; (ii) performs a fourth phaseof deconvolution with a fourth phase regularization spatial mask toreconstruct the texture in at least one of the outlier regions togenerate a fourth phase latent sharp image, the fourth phaseregularization spatial mask being different from the third phaseregularization spatial mask; and (iii) identifying any non-outlierregions in the blurry image; wherein the first phase of deconvolutionreconstructs the main edges in at least one of the non-outlier regions;and wherein the second phase of deconvolution reconstructs the texturein at least one of the non-outlier regions.
 26. The system of claim 25wherein the control system merges the second phase latent sharp imagewith the fourth phase latent sharp image to create the output latentsharp image.
 27. A method for deblurring a blurry image with a known orestimated point spread function, the method comprising the steps of:converting the blurry image to a color space that includes a luminancechannel and a chrominance channel; performing a first phase ofdeconvolution on the luminance channel of the blurry image with a firstlatent sharp image cost function that assumes a Laplacian orHyper-Laplacian prior regularization to generate a luminance channel,first phase latent sharp image having reconstructed main edges;performing a second phase of deconvolution on the luminance channel,first phase latent sharp image with a second latent sharp image costfunction that assumes a Gaussian prior regularization and includes aspatial prior regularization to generate a luminance channel, secondphase latent sharp image with reconstructed texture; performing a firstphase of deconvolution on the chrominance channel of the blurry image toreconstruct the main edges of the chrominance channel; and performing asecond phase of deconvolution on the chrominance channel to reconstructthe texture in edges of the chrominance channel.
 28. A system fordeblurring a blurry image that includes main edges and texture, thesystem comprising: a control system that includes a processor that (i)converts the blurry image to a color space that includes a luminancechannel and a chrominance channel; (ii) performs a first phase ofdeconvolution on the luminance channel of the blurry image with a firstlatent sharp image cost function that assumes a Laplacian orHyper-Laplacian prior regularization to generate a luminance channel,first phase latent sharp image having reconstructed main edges; and (ii)performs a second phase of deconvolution on the luminance channel, firstphase latent sharp image with a second latent sharp image cost functionthat assumes a Gaussian prior regularization and includes a spatialprior regularization to generate a luminance channel, second phaselatent sharp image with reconstructed texture; (iii) performs a firstphase of deconvolution on the chrominance channel of the blurry image toreconstruct the main edges of the chrominance channel; and (iv) performsa second phase of deconvolution on the chrominance channel toreconstruct the texture in edges of the chrominance channel.